Web Scraper

The Web Scraper feature is used to import external data from web sites directly into Aurora. Use the Web Scraper to browse to web sites, select reports or tables, and format them into Aurora CDS tables. This feature even allows scheduling to automatically update data tables and has some capabilities to update the web addresses with date-specific names. The Web Scraper is accessed through the Input Data dropdown button on the main toolbar.

NOTE: The Web Scraper feature is not compatible with all web site data formats. Generally, most easily identified table formats will be recognized, but some web sites may include banners or extraneous information that disrupts the Web Scraper's data converter.

The Web Scraper is accessed through the button on the Tools Tab ribbon.

The Web Scraper can convert most types of data tables including HTML, text, Excel, CSV, XML, and other delimited tables, (also tables in a PDF file or within ZIP folders). You can specify file type, header line number and delimiters (although if not specified, Aurora will attempt to do its best at guessing some of these). You can also specify table name and update parameters, including whether to update the URL for a new date when the table is saved to the CDS. Data can be modified without downloading the original table again.

The Web Scraper form is a tabbed form with three main windows. To begin initial setup for a new table, go to the Web Browser window.

Web Browser - allows the user to navigate to any web page (a handful of power data web pages are predefined). This page also includes buttons used to begin the process of a file download and conversion on the next mouse click.
Web Tables - displays the current table from the set of recently downloaded and converted tables or the CDS (whichever is latest). This page also has a button that allows the downloaded and converted tables to be stored in the CDS.
CDS Tables - displays available created tables, the contents of those tables in the local dataset (CDS), and information related to each table that is saved in extended properties of the table.

The Web Scraper toolbar ribbon provides quick access to CDS files.

Open - This button loads an existing CDS file.

Save & Save As - These buttons save the currently loaded CDS file.

The Scripting command AC.UpdateWebTablesInCDS is also available to update all CDS tables that were created with the Web Scraper.

Productivity Tools

Web Scraper