NVHCloud
Français
Español
Germany
United States
Canada
A site that has just gone live does not show up in Google straight away. The search engine first has to discover it, then crawl its pages, and finally decide to index them. You cannot force this process, but you can make it easier and follow it. This guide covers the technical tools: robots.txt, sitemap and Search Console.
1. How Google discovers a site
Google discovers pages by following links, from other sites or from your own pages, and by reading the sitemaps you submit. Each discovered page is then crawled by Googlebot and evaluated: a page can be crawled without being indexed if Google considers it not very useful, duplicated or low quality. Hence a simple rule: every important page must be reachable through a link from another page on the site, not just listed in the sitemap.
2. The robots.txt file
Placed at the root of the site, robots.txt tells bots which areas not to crawl. A sensible example:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Sitemap: https://www.mysite.com/sitemap.xmlrobots.txt prevents crawling, not indexing. A blocked page can still appear in Google, without a description, if other sites link to it. To remove a page from the index, leave it crawlable and add the <meta name="robots" content="noindex"> tag: Google has to be able to read it to take it into account.The most common mistake after going live is a Disallow: / left over from development, or a noindex tag still on the whole site. Check both on launch day: see the pre-launch SEO checklist.
3. The XML sitemap
The sitemap lists the addresses you want indexed. Most CMSs generate it automatically; for a hand-built site, the format is simple:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://www.mysite.com/</loc>
<lastmod>2026-09-15</lastmod>
</url>
<url>
<loc>https://www.mysite.com/services</loc>
<lastmod>2026-09-01</lastmod>
</url>
</urlset>- Only list indexable pages: final address, 200 response, no
noindex, no redirect. A sitemap full of error or redirected pages loses the search engine's trust. - The
lastmoddate is used by Google if it is reliable: only update it when the content actually changes. - The
priorityandchangefreqtags are ignored by Google: no need to worry about them. - A sitemap is limited to 50,000 addresses and 50 MB. Beyond that, split it and list the parts in a sitemap index.
4. Add the site to Search Console
Google Search Console is the free tool that shows how Google sees your site. Add a Domain property: it covers every variant, with and without www, over HTTP and HTTPS. Verification is done by adding a TXT record provided by Google to the domain's DNS zone: see configuring DNS.
Once the domain is verified, open Sitemaps, enter your sitemap's address and submit. After a few days, the Pages and Performance reports fill up: indexed and non-indexed pages, queries typed by users, clicks and positions.
5. Inspect a page and request indexing
Paste a page's address into the search bar at the top of Search Console. The inspection tool shows whether the page is indexed, when it was crawled, which canonical address Google chose, and lets you test the live URL. For a new or heavily updated page, the Request indexing button puts it in the priority crawl queue. It is limited to a few requests per day: keep it for important pages.
6. Understanding non-indexed pages
| Status in Search Console | Meaning and what to do |
|---|---|
| Discovered – currently not indexed | Google knows the address but has not crawled it yet. Common on a new site; strengthen internal links to the page. |
| Crawled – currently not indexed | Google read it and did not find it useful. Enrich the content, or merge pages that are too similar. |
| Duplicate without user-selected canonical | Several addresses show the same content. Add a canonical tag and redirect the variants. |
| Excluded by 'noindex' tag | Normal if intended; otherwise, remove the tag. |
| Blocked by robots.txt | Check that the rule does not block an important page. |
| Server error (5xx) | The server failed during crawling: monitor its availability and resources. |
A slow response time also slows crawling: Google crawls fewer pages on a server that responds poorly. See optimising PHP-FPM and the Nginx cache.
7. Bing and IndexNow
Also consider Bing Webmaster Tools, which can import your properties directly from Search Console. Bing, like Yandex, supports IndexNow: a protocol that lets you report a new or updated page instantly with a simple request. Several CMS plugins include it. Google does not use it, but the cost is zero and the effect on the engines that support it is immediate.
Frequently asked questions
How long does it take for Google to index a new site?
From a few days to a few weeks. A new site with no external links is crawled slowly. Submit the sitemap in Search Console, request indexing of the main pages and get a few links from existing sites to speed up discovery.
Does the sitemap guarantee indexing?
No. It helps Google discover your pages, but indexing depends on the quality and perceived usefulness of each page. A page listed in the sitemap but with no internal link or substantial content can stay unindexed.
Should I block a page in robots.txt to remove it from Google?
No, it is counterproductive: if Google can no longer crawl the page, it does not see the noindex tag and may keep it in the index. Leave it crawlable with a noindex tag, and use Search Console's removal tool for a quick temporary removal.
Why are my pages "Crawled – currently not indexed"?
Google read them and decided they did not add enough value compared with what already exists. The most common causes are content that is too short, pages very similar to one another, or a site that is still little known. Enrich these pages and link to them from your strongest pages.