[Bucardo-general] fsm and vm files generated under $PGDATA/base

[akp geek]

Dear all -

During the replication process , files are getting generated in
$PGDATA/base directory. Some of the file end up with _fsm and some _vm. They
are consuming lot of space on the server. Can you please share your thoughts
on these.


Regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mail.endcrypt.com/pipermail/bucardo-general/attachments/20100302/fa7d9385/attachment.html

[Rosser Schwarz]

???????????? During the replication process , files are getting generated in
$PGDATA/base directory. Some of the file end up with _fsm and some _vm.

The *_fsm files are used for a relation's Free Space Map, which tracks
... free space in the relation (tuples/pages which are known dead and
re-usable). The *_vm files store the Visibility Map, which is a list
of pages containing only tuples visible to all transactions. They're
generated and used by Postgres' autovacuum daemon. Like the *.1, *.2
... *.N files that Postgres creates to partition tables' disk files at
one gbyte boundaries, you'll want to leave them alone.

See sections 53.3 and 53.4 of TFM for 8.4 for further details.

rls

[akp geek]

The only reason I got concerned is after I started the replication process,
the size of my $PGDATA on master is about 849M and on the slave is 7.8G.
It's growing tremondously. I am not able to figure out why.

All I have one master is bucardo and the database that I am trying to
replicate. That's it. any clue why the size of base directory is getting
increased?

Regards

Post by Rosser Schwarz

Post by akp geek
During the replication process , files are getting generated

in

Post by akp geek
$PGDATA/base directory. Some of the file end up with _fsm and some _vm.

The *_fsm files are used for a relation's Free Space Map, which tracks
... free space in the relation (tuples/pages which are known dead and
re-usable). The *_vm files store the Visibility Map, which is a list
of pages containing only tuples visible to all transactions. They're
generated and used by Postgres' autovacuum daemon. Like the *.1, *.2
... *.N files that Postgres creates to partition tables' disk files at
one gbyte boundaries, you'll want to leave them alone.
See sections 53.3 and 53.4 of TFM for 8.4 for further details.
rls
--
:wq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mail.endcrypt.com/pipermail/bucardo-general/attachments/20100302/4bb68826/attachment-0001.html

[akp geek]

Hi All -


I have turned on the auto vacuum on the slave still for
some reason the db is getting bloated up. The master size is only 1G and the
slave is at 9.2GB now. I did cluster on couple of tables also. did any one
run into this situation? Can you please help?

Regards

Post by akp geek
The only reason I got concerned is after I started the replication
process, the size of my $PGDATA on master is about 849M and on the slave is
7.8G. It's growing tremondously. I am not able to figure out why.
All I have one master is bucardo and the database that I am trying to
replicate. That's it. any clue why the size of base directory is getting
increased?
Regards

Post by akp geek

Post by akp geek
During the replication process , files are getting

generated in

Post by akp geek
$PGDATA/base directory. Some of the file end up with _fsm and some _vm.

The *_fsm files are used for a relation's Free Space Map, which tracks
... free space in the relation (tuples/pages which are known dead and
re-usable). The *_vm files store the Visibility Map, which is a list
of pages containing only tuples visible to all transactions. They're
generated and used by Postgres' autovacuum daemon. Like the *.1, *.2
... *.N files that Postgres creates to partition tables' disk files at
one gbyte boundaries, you'll want to leave them alone.
See sections 53.3 and 53.4 of TFM for 8.4 for further details.
rls
--
:wq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mail.endcrypt.com/pipermail/bucardo-general/attachments/20100305/d22529a7/attachment.html

[Rosser Schwarz]

??????????????? I have turned on the auto vacuum on the slave?still ?for
some reason the db is getting bloated up.

It isn't just a matter of "turn[ing] on the auto vacuum". How many of
your tables have entries in the slave node's pg_autovacuum system
catalog?

rls

[akp geek]

Thank you. I am using postgres 8.4 and noticed that pg_autovacuum table is
not there. I will try to find alternatives for it , means if the data is
getting stored some other catalog tables. I will update

Regards

Post by Rosser Schwarz

Post by akp geek
I have turned on the auto vacuum on the slave still for
some reason the db is getting bloated up.

It isn't just a matter of "turn[ing] on the auto vacuum". How many of
your tables have entries in the slave node's pg_autovacuum system
catalog?
rls
--
:wq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mail.endcrypt.com/pipermail/bucardo-general/attachments/20100305/0981e8b4/attachment.html

[Rosser Schwarz]

Post by akp geek
Thank you. I am using postgres 8.4 and noticed that pg_autovacuum table is
not there.

D'oh; I should have inferred that from the file names. That table
only exists in releases prior to 8.4. Please consult TFM,
particularly section 23.1.5, and make sure you have everything
configured properly on the slave to support autovacuum.

rls

[akp geek]

Sure I will check. Thanks again for the help

Regards

Post by Rosser Schwarz

Post by akp geek
Thank you. I am using postgres 8.4 and noticed that pg_autovacuum table

is

Post by akp geek
not there.

D'oh; I should have inferred that from the file names. That table
only exists in releases prior to 8.4. Please consult TFM,
particularly section 23.1.5, and make sure you have everything
configured properly on the slave to support autovacuum.
rls
--
:wq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mail.endcrypt.com/pipermail/bucardo-general/attachments/20100305/bbbbc19c/attachment.html

[Greg Sabino Mullane]

Post by akp geek
I have turned on the auto vacuum on the slave still
for some reason the db is getting bloated up. The master size is only
1G and the slave is at 9.2GB now. I did cluster on couple of tables
also. did any one run into this situation? Can you please help?

This is a little out of the purview of this list, but...

* Existing bloat:

You can use check_postgres.pl --action=bloat to see which tables are the
most bloated, or simply start suspecting the largest ones by running:

SELECT relname, pg_size_pretty(pg_total_relation_size(oid))
FROM pg_class
WHERE relkind = 'r'
ORDER BY pg_total_relation_size(oid) DESC LIMIT 10;

Once you find a suspect table, you can pg_relation_size('tablename')
and pg_relation_size('indexname') to see how big the main table and
indexes are. Try vacuuming them (or cluster) and see if they shrink.
If the indexes are large, try REINDEX on them.

* Future bloat:

If you are doing fullcopy syncs, try the truncate mode versus the
delete one for some long-term bloat savings.

If the tables are filling up too quick, you can tell autovacuum to be
more aggressive. The defaults in postgresql.conf are very conservative.
In addition, you can adjust them per table with pg_autovacuum < 8.4,
and ALTER TABLE .. SET in 8.4 and up. See:

http://www.postgresql.org/docs/8.4/static/sql-createtable.html
--
Greg Sabino Mullane greg at endpoint.com
End Point Corporation
PGP Key: 0x14964AC8
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 163 bytes
Desc: not available
Url : https://mail.endcrypt.com/pipermail/bucardo-general/attachments/20100305/7fadbbfe/attachment.bin

[akp geek]

I have to really agree with you. I really appreicate your time for
responding this. I am getting better knowledge on the bucardo and also on
postgres with all your support.

I will take your suggestions. I will put the
check_postgres.pl--action=bloatin our nagios and monitor it and take
necessary actions

again thanks for the help

Regards

Post by Greg Sabino Mullane

Post by akp geek
I have turned on the auto vacuum on the slave still
for some reason the db is getting bloated up. The master size is only
1G and the slave is at 9.2GB now. I did cluster on couple of tables
also. did any one run into this situation? Can you please help?

This is a little out of the purview of this list, but...
You can use check_postgres.pl --action=bloat to see which tables are the
SELECT relname, pg_size_pretty(pg_total_relation_size(oid))
FROM pg_class
WHERE relkind = 'r'
ORDER BY pg_total_relation_size(oid) DESC LIMIT 10;
Once you find a suspect table, you can pg_relation_size('tablename')
and pg_relation_size('indexname') to see how big the main table and
indexes are. Try vacuuming them (or cluster) and see if they shrink.
If the indexes are large, try REINDEX on them.
If you are doing fullcopy syncs, try the truncate mode versus the
delete one for some long-term bloat savings.
If the tables are filling up too quick, you can tell autovacuum to be
more aggressive. The defaults in postgresql.conf are very conservative.
In addition, you can adjust them per table with pg_autovacuum < 8.4,
http://www.postgresql.org/docs/8.4/static/sql-createtable.html
--
Greg Sabino Mullane greg at endpoint.com
End Point Corporation
PGP Key: 0x14964AC8
_______________________________________________
Bucardo-general mailing list
Bucardo-general at bucardo.org
https://mail.endcrypt.com/mailman/listinfo/bucardo-general

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://mail.endcrypt.com/pipermail/bucardo-general/attachments/20100305/c3a71fdb/attachment-0001.html