Optimize Script

thiagotgc

Active Member
Dec 17, 2019
151
20
38
37
I created a shell script, so that every day the data from another server with the email lists are imported into the PMG.
So every day a script runs causing the domains to be deleted and re-added in the PMG.

Code:
echo "Clearing current Blacklist of incoming emails"
pmgsh get /config/ruledb/who/83/objects | sort |grep id | cut -d : -f 2- | sed 's/[ \t]*//;s/[ \t]*"//' | sed 's/[ \t]*//;s/[ \t]*"//' | sed 's/[ \t]*//;s/[ \t]*,//' > /tmp/del_list
for DEL in `cat /tmp/del_list`; do pmgsh delete /config/ruledb/who/83/objects/$DEL; done >/dev/null 2>&1


echo "Importing Blacklist of Incoming Emails to PMG"
while read blacklist ; do pmgsh create /config/ruledb/who/83/email --email "$blacklist" ; done < /tmp/pmg_blacklist_emails >/dev/null 2>&1

Everything runs correctly, and I'm getting the process I need, but the list is getting bigger every day, and it's taking hours to delete and hours to add.

How could I optimize for the pmgsh call to be unique, without making it open its own shell?
 
Hi.
I think you need compare blacklist for 2 days, find diff and delete\import only it.
Example:
cat blacklist_05_29_23
baddomain1.com
baddomain2.com

cat blacklist_05_30_23
baddomain3.com
baddomain2.com

If we find diff beetween 29 & 30
diff blacklist_05_30_23 blacklist_05_29_23 | grep '^>' | cut -c 3-
baddomain1.com
we find domain, that was in old BL & not present in current BL, so we only can delete it via pmgsh delete /config/ruledb/who/83/objects/$DEL; done >/dev/null 2>&1

If we find diff beetween 30 & 29
diff blacklist_05_29_23 blacklist_05_30_23 | grep '^>' | cut -c 3-
baddomain3.com
we find new domain in BL, so we can import only it via pmgsh create /config/ruledb/who/83/email --email "$blacklist"

So you must save pmg_blacklist_emails with date, compare it and only delete/import diff.
 
Hi.
I think you need compare blacklist for 2 days, find diff and delete\import only it.
Example:
cat blacklist_05_29_23
baddomain1.com
baddomain2.com

cat blacklist_05_30_23
baddomain3.com
baddomain2.com

If we find diff beetween 29 & 30
diff blacklist_05_30_23 blacklist_05_29_23 | grep '^>' | cut -c 3-
baddomain1.com
we find domain, that was in old BL & not present in current BL, so we only can delete it via pmgsh delete /config/ruledb/who/83/objects/$DEL; done >/dev/null 2>&1

If we find diff beetween 30 & 29
diff blacklist_05_29_23 blacklist_05_30_23 | grep '^>' | cut -c 3-
baddomain3.com
we find new domain in BL, so we can import only it via pmgsh create /config/ruledb/who/83/email --email "$blacklist"

So you must save pmg_blacklist_emails with date, compare it and only delete/import diff.

thanks for the idea

Using diff to add new domains, ok, but to delete it is difficult, because I would need to collect the ID of each domain, and then delete it.
 
Before deleting you can dump pmg conf via
Bash:
pmgdb dump > /temp/config
find ID of BL object via command
Bash:
cat /temp/config|grep baddomain1.com|tr -d -c 0-9
and than remove it via
Bash:
pmgsh delete /config/ruledb/who/83/objects/$DEL

So final script must look like:
Bash:
echo "Find diffs, that must be deleted"
# find diffs, that not exist in new BL
diff blacklist_05_30_23 blacklist_05_29_23 | grep '^>' | cut -c 3- > /tmp/bl_to_remove
echo "Dump PMG config"
# dump current config
pmgdb dump > /temp/config
echo "Remove old objects"
# find id of object and remove it
for DEL in `cat /tmp/bl_to_remove`; do id=$(cat /temp/config|grep $DEL|tr -d -c 0-9);pmgsh delete /config/ruledb/who/83/objects/$id; done >/dev/null 2>&1
echo "Find diffs, that must be imported"
# find diffs, that must be added
diff blacklist_05_29_23 blacklist_05_30_23 | grep '^>' | cut -c 3- > /tmp/bl_to_add
echo "Importing new items"
# add new item
while read blacklist ; do pmgsh create /config/ruledb/who/83/email --email "$blacklist" ; done < /tmp/bl_to_add >/dev/null 2>&1
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!