One of the questions that we are often asked during our live MCLE seminars and MCLE webinars is whether there is a keyword search available for the billions of Web pages collected by the Archive.org's Wayback Machine. Since its inception in 2000, Wayback Machine users have been able to browse earlier versions of websites by entering the URL for a known site into the main search box and clicking on Browse History.

Even though the Archive.org has attempted to develop a keyword for its collection search several times, none ever worked well-enough to become a permanent addition. So, the answer has always been, "No, there is no keyword search of the Web pages stored in the Internet Archive's Wayback Machine," ...until now...sort of. Archive.org has recently added a new keyword search through a portion of its collection.

The Internet Archive announced the keyword search feature in a blog post that included the following:

With this new beta search service, users will now be able to find the home pages of over 361 Million websites preserved in the Wayback Machine just by typing in keywords that describe these sites (e.g. “new york times”). As they type keywords into the search box, they will be presented with a list of relevant archived websites with snippets containing:

• a link to the archived versions of the site’s home page in the Wayback Machine
• a thumbnail image of the site’s homepage (when available)
• a short description of the site’s homepage
• a capture summary of the site
  • number of unique URLs by content type (webpage, image, audio, video)
  • number of valid web captures over the associated time period

 

Key Features

• Search as you type
  • Instant results as you type — predictive, interactive and speedy
• Multilingual
  • Search in any language or using symbols — expanding scope and utility
• Site-based Filtering
  • Limit results to certain websites or domains using the site: operator (e.g. site:edu)

One important limitation of this new search is that it searches ONLY text on the home pages of the sites and NOT all of the pages of sites contained in the Wayback Machine.

It's interesting to note that while the blog post refers to the site's ability to keyword search "the home pages of over 361 Million websites," the main search box currently offers visitors the ability to "Search the history of over 279 billion web pages on the Internet" using a specific URL. Prior to the introduction of the new home page keyword search, the site previously boasted nearly 475 billion Web pages in its collection. While the difference in the number of browseable historic pages may be explained by the distinction Archive.org makes between "Web pages, Web sites, and Web captures," we are awaiting a response from the organization to confirm.

The post Internet Archive’s Wayback Machine Launches Full-Text Search…Sort Of appeared first on Continuing Legal Education (MCLE) in California, MCLE Los Angeles, CLE Orange County, Irvine & San Francisco.

The Internet Archive gives us access to old versions of pages that have been collected and stored on its servers. Subsequent changes made to the "live" version of the Web page do not effect the version stored in the Archive's collection. Searching the Archive gives us the ability to see what Web pages looked like in the past.

Archived Copies of the Internet For Lawyers Web site from Archive.org

May 11, 2000	September 30, 2005

While the Internet Archive collection wasn't launched publicly until 2001, it actually began collecting Web pages in 1996. Unlike some of the traditional search engines that offer access only to the most recent cached versions of the old Web pages they’ve captured, the Way Back Machine offers access to all of the versions of any Web page it has captured. To locate any of these old pages, you must enter the URL of the specific site (or page) you’re interested in locating into the search box on Archive.org’s home page and click the "Browse History" button. (It is important to note, however, that you cannot do a keyword search through this collection of pages)

The search results will display a calendar list of the most recent captures of the URL searched for that are available in the Archive.org collection. Each blue dot on the calendar represents a day on which the searched URL was crawled. The larger the blue dot is, the more snapshots of content from that URL were captured on that day. Clicking on any of the blue dots will display that old Web page as it appeared on the date selected.

The timeline at the top of the calendar graphically depicts the distribution of all captures for the searched URL. Clicking on any of the years in the timeline will then display all of the captures for the searched URL for the year selected. The timeline also lists the number of times the searched URL has been crawled by Archive.org and the earliest date on which the URL was crawled. Note, however, that in our test searches clicking on a year where no captures were displayed in the timeline occasionally returned a cached page from Archive.org’s collection.

The post Internet Archive Now Contains 400 Billion Webpages appeared first on Continuing Legal Education (MCLE) in California, MCLE Los Angeles, CLE Orange County, Irvine & San Francisco.

The Internet Archive has launched a beta version of a new search interface and results display scheme for the Wayback Machine.

The Wayback Machine beta version's Latest button shows only the most recent version of the searched-for URL from the Internet Archive's collection. The Show All button is the same as the classic Wayback Machine's Take Me Back button.

The two most noticeable differences are the way the pages (and their harvesting dates) are represented with a timeline and calendar, and the addition of a toolbar at the top of pages displayed from the collection.

Results of a URL search in the Wayback Machine's collection of archival Web pages now shows the harvesting dates of the pages distributed on a calendar.

The new toolbar at the top of the pages displayed from the Wayback Machine makes it easier to quickly see the date on which that particular page was collected.

According to the site's Frequently Asked Questions, the beta is also built to be faster than the "classic" Wayback Machine search. 

The Internet Archive's Wayback Machine is a historical collection of Web pages (as they appeared at specific points in time). The Wayback Machine's collection dates back to 1996 (although not all sites in the collection are represented back that far). The public search interface was launched in 2001 with a collection of 10 billion pages. According to the Internet Archive, the Wayback Machine's collection now includes 150 billion pages.

The post Internet Archive Launches New Beta Version of the Wayback Machine appeared first on Continuing Legal Education (MCLE) in California, MCLE Los Angeles, CLE Orange County, Irvine & San Francisco.

One of the most popular resources that we discuss in our live Continuing Legal Education seminars using the Internet for investigative research is the Internet Archive (aka Archive.org). Numerous attorneys have told us that the site's collection of old versions of billions of Web pages has provided important information in the matters they've handled. (Still more tell us that they wish they were aware of it previously because it might have been useful to them in a previous matter.) 

The pages collected by the Internet Archive are not immediately available for retrieval however. There has always been a lag time of six months between the date on which copies of Web pages were collected and when they became available as part of the searchable collection. Recently however, we have had trouble locating versions of Web pages in the Archive collection that are newer than 18 months. Note that in the screenshot below, a search for archived versions of pages from this site (conducted on November 10, 2010) yields no pages newer than June 25, 2008.

The site's Frequently Asked Questions (FAQ) page has been revised to note a longer lag time than the previously cited six months.

It generally takes 6 months or more (up to 24 months) for pages to appear in the Wayback Machine after they are collected, because of delays in transferring material to long-term storage and indexing, or the requirements of our collection partners. In some cases, crawled content from certain projects can appear in a much shorter timeframe — as little as a few weeks from when it was crawled. Older material for the same pages and sites may still appear separately, months later. There is no access to files before they appear in the Wayback Machine.

However, it should also be noted that another portion of their FAQ states, "Please note that there is a 6 - 14 month lag time."  

Either way, it means that we will be waiting longer for these archived versions of Web pages to be available to us.

The post Internet Archive Increases Delay in Posting Pages to Collection appeared first on Continuing Legal Education (MCLE) in California, MCLE Los Angeles, CLE Orange County, Irvine & San Francisco.