What is website indexing? How does it happen? You can find answers to these and other questions in the article. Web indexing (indexing in search engines) is the process of adding information about a site to the database by a search engine robot, which is subsequently used to search for information on web projects that have undergone such a procedure.
Data about web resources most often consists of keywords, articles, links, documents. Audio, images, and so on can also be indexed. It is known that the keyword detection algorithm is dependent on the search engine.
There is some limitation on the types of indexed information (flash files, javascript).
Initiation management
Indexing a site is a complex process. To manage it (for example, to prohibit the attachment of a particular page), you need to use the robots.txt file and such instructions as Allow, Disallow, Crawl-delay, User-agent and others.
Also, tags and props are used for indexing, hiding the content of the resource from Google and Yandex robots (Yahoo uses the tag).
In the Goglle search engine, new sites are indexed from a couple of days to one week, and in Yandex - from one to four weeks.
Do you want your site to appear in search engine results queries? Then it must be processed by Rambler, Yandex, Google, Yahoo, and so on. You must inform search engines (spiders, systems) about the existence of your website, and then they will crawl it in whole or in part.
Many sites have not been indexed for years. The information that is on them is not seen by anyone except their owners.
Processing methods
Indexing a site can be done in several ways:
- First option is manual addition. You need to enter your site data through special forms offered by search engines.
- In the second case, the search engine robot itself finds your web site by links and indexes it. He can find your site by links from other resources that lead to your project. This method is the most efficient. If a search engine finds a site in this way, it considers it significant.
Timing
Indexing the site is not too fast. Terms vary, from 1-2 weeks. Links from authoritative resources (with excellent PR and Titz) significantly speed up the placement of the site in the search engine database. Today, Google is considered the slowest, although until 2012 it could do this job in a week. ToUnfortunately, things change very quickly. It is known that Mail.ru has been working with websites in this area for about six months.
Indexing a site in search engines is not possible for every specialist. The timing of adding new pages to the database of a site already processed by search engines is affected by the frequency of updating its content. If fresh information constantly appears on a resource, the system considers it frequently updated and useful to people. In this case, her work is accelerated.
You can follow the indexing progress of a web site on special sections for webmasters or on search engines.
Changes
So, we have already figured out how the site is indexed. It should be noted that search engine databases are frequently updated. Therefore, the number of pages of your project added to them may change (both decrease and increase) for the following reasons:
- search engine sanctions against the website;
- the presence of errors on the site;
- changing search engine algorithms;
- disgusting hosting (inaccessibility of the server where the project is located) and so on.
Yandex answers to common questions
"Yandex" is a search engine used by many users. It ranks fifth among the world's search systems in terms of the number of processed research requests. If you added a site to it, it may take too long to add to the database.
Adding a URL does not guarantee its indexing. This is just one of the methods by which the system robot is toldabout a new resource. If there are few or no links to a site from other websites, adding it will help you find it faster.
If indexing did not occur, you need to check if there were any failures on the server at the time of creating an application for it from the Yandex robot. If the server reports an error, the robot will terminate its work and try to complete it in a round trip order. Yandex employees cannot increase the speed of adding pages to the search engine database.
Indexing a site in Yandex is a rather difficult task. You do not know how to add a resource to a search engine? If there are links to it from other websites, then you do not need to add a special site - the robot will automatically find it and index it. If you don't have such links, you can use the "Add URL" form to tell the search engine that the site exists.
Keep in mind that adding a URL does not guarantee your creation will be indexed (or indexed).
Many people wonder how long it takes to index a site in Yandex. Employees of this company do not give guarantees and do not predict terms. As a rule, since the robot found out about the site, its pages in the search appear in two days, sometimes in a couple of weeks.
Process
"Yandex" is a search engine that requires accuracy and attention. Site indexing consists of three parts:
- Search robot crawls resource pages.
- Content(content) of the site is recorded in the database (index) of the search system.
- In 2-4 weeks, after updating the database, you can see the results. Your site will (or won't) appear in search results.
Indexing check
How to check website indexing? There are three ways to do this:
- Enter the name of your business in the search bar (for example, "Yandex") and check every link on the first and second pages. If you find the URL of your brainchild there, then the robot has completed its task.
- You can enter your website URL in the search bar. You will be able to see how many internet sheets are shown, i.e. indexed.
- Register on the pages of webmasters in Mail.ru, Google, Yandex. After you pass the site verification, you will be able to see the indexing results and other search engine services created to improve the performance of your resource.
Why does Yandex fail?
Indexing a site in Google is carried out as follows: the robot enters into the database all the pages of the site, low-quality and high-quality, without selecting. But only useful documents participate in the ranking. And "Yandex" immediately excludes all web trash. It can index any page, but the search engine will eventually eliminate all garbage.
Both systems have an additional index. Both low quality pages affect the ranking of the web site as a whole. There is a simple philosophy at work here. Favorite resources of a particularthe user will occupy higher positions in his issuance. But this same individual will have a hard time finding a site that he did not like last time.
That is why, first, it is necessary to cover copies of web documents from indexing, check for empty pages and prevent low-quality content from being indexed.
Speed up Yandex
How can I speed up site indexing in Yandex? Follow these steps:
- Install the Yandex browser on your computer and use it to browse the pages of the site.
- Confirm the rights to manage the resource in Yandex. Webmaster.
- Post a link to the article on Twitter. It is known that Yandex has been cooperating with this company since 2012.
- Add search from Yandex for the site. In the "Indexing" section, you can enter your own URLs.
- Enter the "Yandex. Metrica" code without ticking "Submitting pages for indexing is prohibited".
- Produce a Sitemap that exists only for the robot and is not visible to the audience. Verification will begin with him. The Sitemap address is entered in robots.txt or in the appropriate form in the "Webmaster" - "Indexing Settings" - "Sitemap Files".
Intermediate actions
What needs to be done until the web page is indexed by Yandex? The domestic search engine should consider the site as the primary source. That is why even before the publication of the article, it is imperative to add its content to the form of "Specific texts". Otherwiseplagiarists will copy the record to their resource and will be the first in the database. As a result, they will be recognized as the authors.
Google Database
For Google, the same recommendations that we described above are suitable, only the services will be different:
- Google+ (replacing Twitter);
- Google Chrome;
- Google Tools for Programmers - "Scan" - "Look like Googlebot" - option "Scan" - option "Index";
- search within a resource from Google;
- Google Analytics (instead of Yandex. Metrics).
Prohibition
What is a site indexing ban? You can overlay it both on the entire page and on a separate part of it (link or piece of text). In fact, there is both a global indexing ban and a local one. How is it implemented?
Let's consider the prohibition of adding a web site to the search engine database in Robots.txt. Using the robots.txt file, you can exclude the indexing of one page or an entire resource heading like this:
- User-agent:
- Disallow: /kolobok.html
- Disallow: /foto/
The first point says that the instructions are defined for all PSs, the second one indicates that the indexing of the kolobok.html file is prohibited, and the third one does not allow adding the entire stuffing of the foto folder to the database. If you need to exclude multiple pages or folders, please specify them all in Robots.
In order to prevent the indexing of a particular Internet sheet, you can use the robots meta tag. It is different from robots.txtthe fact that it gives instructions to all the PS at once. This meta tag follows the general principles of the html format. It should be placed in the title of the page between the tags. An entry for a ban, for example, might be written like this:.
Ajax
How does Yandex index Ajax sites? Today, Ajax technology is used by many web site developers. Of course, she has great potential. With it, you can create fast and productive interactive web pages.
However, the search engine robot "sees" the web list differently than the user and the browser. For example, a person looks at a comfortable interface with movably loaded Internet sheets. For a crawler, the content of the same page can be empty or presented as the rest of the static HTML content, for which scripts do not go to work.
You can use a URL withto create Ajax sites, but the search engine does not use it. Usually the part of the URL after theis separated. This must be taken into account. Therefore, instead of a URL like https://site.ru/example, he makes an application to the main page of the resource located at https://site.ru. This means that the content of the Internet sheet may not get into the database. As a result, it will not appear in the search results.
To improve the indexing of Ajax sites, Yandex supported changes in the search robot and the rules for processing the URLs of such web sites. Today, webmasters can indicate to the Yandex search engine the need for indexing by creating an appropriate scheme in the resource structure. For this you need:
- Replace the symbolin the URL of the pageson the !. Now the robot will understand that it can apply for the HTML version of the content of this Internet sheet.
- HTML version of the content of such a page should be placed on a URL where ! replaced with ?_escaped_fragment_=.
