2MASS Catalog Server Kit

Install Huge Catalogs


Page Index:
HOME / Install Huge Catalogs / Install Small Catalogs / Information about table columns / Hard Drive Copy Service


Before Installing Catalogs

Before installing catalogs, you have to setup OS, PostgreSQL and 2MASS Kit following the installation procedure.

Installing 2MASS PSC

We show the procedure of installing 2MASS PSC. You can also install other huge catalogs supported in this kit with instructions in this page. Replace `twomass' when installing other catalogs.

Download data files

Download all required data files and then extract them all.

Here is an example using wget command:

$ mkdir twomass_data
$ cd twomass_data
$ wget -nd -r http://darts.jaxa.jp/pub/ir/2masskit/v2/twomass/
$ gunzip *.db.txt.gz

Creating Symbolic Links

In `twomass' directory of 2MASS Kit, create symbolic links to downloaded data files.

$ cd twomass
$ ln -s (somewhere)/twomass_data/*.db.txt .

Setting-up SQL Files

Edit the Makefile:

$ vi Makefile

Two tables are created for 2MASS PSC: "Main Table," and "Dedicated Positional Search Table." Basically all positional searches use the Dedicated Positional Search Table via stored functions. With other searches the vanilla SELECT ... FROM ... WHERE ... syntax is used for searching the Main Table. However, if you don't specify coordinates and force a search with flux or the like you will not be able to search at a reasonable speed, even if you did create an index, unless the search space is narrowed down to several thousand near the center of the distribution using a "LIMIT" clause.

In the Makefile, INDICES specifies which columns in Main Table are populated with indices. The default is j_m,h_m,k_m, which creates indices for flux j, h, and k. Add column names as needed. However, do not add objid because an index is created for it even if unspecified.

And then, run make.

$ make

SQL files will be generated.

Registering table spaces for PostgreSQL

As a normal user in UNIX log in to the database "2MASS" as 'postgres' (superuser in PostgreSQL) to register the table spaces:

$ psql -U postgres 2MASS
Password for user postgres: xxxxxx
psql (8.4.5)
Type "help" for help.

2MASS=# \i twomass_create_tablespace.sql 
CREATE TABLESPACE
CREATE TABLESPACE
CREATE TABLESPACE
CREATE TABLESPACE
CREATE TABLESPACE
CREATE TABLESPACE
2MASS=# \q

Registering Parent Tables

As a normal user in UNIX log in to the database "2MASS" as 'admin' to register tables:

$ psql -U admin 2MASS
Password for user admin: xxxxxx
psql (8.4.5)
Type "help" for help.

2MASS=> \i twomass_create_table.sql 
CREATE TABLE
CREATE TABLE

Note: From here on in we will be using commands that take quite a long time to execute. I will include the time required with an Adaptec RAID 2405 + S-ATA2 HDD x 2 (RAID 1), but twice to ten times longer could be required depending on the environment used.

Registering Data in the Main Table

The next step involving registering data and creating the primary keys in the Main Table. This will take several hours (five hours or more) to complete:

2MASS=> \i twomass_main_copy.sql 
SET
psql:twomass_main_copy.sql:xxxxxxxxx: NOTICE:  ALTER TABLE / ADD PRIMARY KEY will create implicit index "twomass_pkey" for table "twomass"
ALTER TABLE
SET

The following type of message may occasionally seen, which can be safely ignored:

LOG:  checkpoints are occurring too frequently (7 seconds apart)
HINT:  Consider increasing the configuration parameter "checkpoint_segments".

If the data registration process does not finish within three days or so you may have some problematic BIOS settings with the S-ATA interface. Refer to FAQ section for more details.

Creating Dedicated Positional Search Table

Create the table, register the data, and create indices as shown below. It will take about 30 minutes to register the data and 80 minutes to create the indices:

2MASS=> \i twomass_eqi_create_table.sql
2MASS=> \i twomass_eqi_copy.sql
2MASS=> \i twomass_eqi_create_index.sql

The floowing type of message may occasionally be seen during the index creation, which can be safely ignored:

ERROR:  canceling autovacuum task
CONTEXT:  automatic analyze of table "xxxxxxxx"

Registering Stored Functions for 2MASS

Register stored functions for 2MASS:

2MASS=> \i twomass_create_function.sql

Granting Table Access to 'guest' Role

2MASS=> \i twomass_main_grant.sql
2MASS=> \i twomass_eqi_grant.sql

Create Index of Main Table (Optional)

If you are using something other than coordinates as the top priority search conditions of you may wish to create indices in the Main Table. It will take several hours to complete with standard settings.

2MASS=> \i twomass_main_create_index.sql

Indices can be added to the Main Table later if preferred. After adding column names to INDICES in the Makefile run, make and then execute "twomass_main_create_index.sql" as above. In this case ensure to run "VACUUM ANALYZE Twomass;" after the indices have been created.

VACUUM

At long last we have arrived at the end of our command sequence. Enter this command and wait a few hours, upon which you should then be able to perform searches. Congratulations!

2MASS=> \i twomass_vacuum.sql

During the execution of VACUUM ANALYZE without arguments you will see a lot of messages like "WARNING: skipping "pg_type" --- only superuser or database owner can vacuum it", but you can safely ignore them.