Optimize Script

[thiagotgc]

I created a shell script, so that every day the data from another server with the email lists are imported into the PMG.
So every day a script runs causing the domains to be deleted and re-added in the PMG.

echo "Clearing current Blacklist of incoming emails"
pmgsh get /config/ruledb/who/83/objects | sort |grep id | cut -d : -f 2- | sed 's/[ \t]*//;s/[ \t]*"//' | sed 's/[ \t]*//;s/[ \t]*"//' | sed 's/[ \t]*//;s/[ \t]*,//' > /tmp/del_list
for DEL in `cat /tmp/del_list`; do pmgsh delete /config/ruledb/who/83/objects/$DEL; done >/dev/null 2>&1


echo "Importing Blacklist of Incoming Emails to PMG"
while read blacklist ; do pmgsh create /config/ruledb/who/83/email --email "$blacklist" ; done < /tmp/pmg_blacklist_emails >/dev/null 2>&1

Everything runs correctly, and I'm getting the process I need, but the list is getting bigger every day, and it's taking hours to delete and hours to add.

How could I optimize for the pmgsh call to be unique, without making it open its own shell?

 

[kalachev_89]

Hi.
I think you need compare blacklist for 2 days, find diff and delete\import only it.
Example:
cat blacklist_05_29_23
baddomain1.com
baddomain2.com

cat blacklist_05_30_23
baddomain3.com
baddomain2.com

If we find diff beetween 29 & 30
diff blacklist_05_30_23 blacklist_05_29_23 | grep '^>' | cut -c 3-
baddomain1.com
we find domain, that was in old BL & not present in current BL, so we only can delete it via pmgsh delete /config/ruledb/who/83/objects/$DEL; done >/dev/null 2>&1

If we find diff beetween 30 & 29
diff blacklist_05_29_23 blacklist_05_30_23 | grep '^>' | cut -c 3-
baddomain3.com
we find new domain in BL, so we can import only it via pmgsh create /config/ruledb/who/83/email --email "$blacklist"

So you must save pmg_blacklist_emails with date, compare it and only delete/import diff.

 

[thiagotgc]

Hi.
I think you need compare blacklist for 2 days, find diff and delete\import only it.
Example:
cat blacklist_05_29_23
baddomain1.com
baddomain2.com

cat blacklist_05_30_23
baddomain3.com
baddomain2.com

If we find diff beetween 29 & 30
diff blacklist_05_30_23 blacklist_05_29_23 | grep '^>' | cut -c 3-
baddomain1.com
we find domain, that was in old BL & not present in current BL, so we only can delete it via pmgsh delete /config/ruledb/who/83/objects/$DEL; done >/dev/null 2>&1

If we find diff beetween 30 & 29
diff blacklist_05_29_23 blacklist_05_30_23 | grep '^>' | cut -c 3-
baddomain3.com
we find new domain in BL, so we can import only it via pmgsh create /config/ruledb/who/83/email --email "$blacklist"

So you must save pmg_blacklist_emails with date, compare it and only delete/import diff.

thanks for the idea

Using diff to add new domains, ok, but to delete it is difficult, because I would need to collect the ID of each domain, and then delete it.

 

[kalachev_89]

Before deleting you can dump pmg conf via

pmgdb dump > /temp/config

find ID of BL object via command

cat /temp/config|grep baddomain1.com|tr -d -c 0-9

and than remove it via

pmgsh delete /config/ruledb/who/83/objects/$DEL

So final script must look like:

echo "Find diffs, that must be deleted"
# find diffs, that not exist in new BL
diff blacklist_05_30_23 blacklist_05_29_23 | grep '^>' | cut -c 3- > /tmp/bl_to_remove
echo "Dump PMG config"
# dump current config
pmgdb dump > /temp/config
echo "Remove old objects"
# find id of object and remove it
for DEL in `cat /tmp/bl_to_remove`; do id=$(cat /temp/config|grep $DEL|tr -d -c 0-9);pmgsh delete /config/ruledb/who/83/objects/$id; done >/dev/null 2>&1
echo "Find diffs, that must be imported"
# find diffs, that must be added
diff blacklist_05_29_23 blacklist_05_30_23 | grep '^>' | cut -c 3- > /tmp/bl_to_add
echo "Importing new items"
# add new item
while read blacklist ; do pmgsh create /config/ruledb/who/83/email --email "$blacklist" ; done < /tmp/bl_to_add >/dev/null 2>&1