Our Puppet enterprise server /var folder ran out of space during the weekend. There were 17123 pending tasks to be waiting in the queue. Refer following,
After increasing the LUN size we have rebooted the server but couldn't get Puppet dashboard to function normally and pending tasks message continue to appear. Did some researching and found this
Console Maintenance Tasks but with no avail. The reason is it wont allow to restart the "pe-puppet-dashboard-workers" and giving me following error,
Stopping pe-dashboard-workers: Terminated
If you are like me here is the quick solution,
Go to following location,
# cd /var/opt/cache/pe-puppet-dashboard/pids
Then delete (Keep a copy if you want to run them or investigate it further) all the pids,
# rm -fr *
Restart Puppet Enterprise and worker modules,
# service pe-httpd restart
# service pe-puppet-dashboard-workers restart
Then should restart normally. That's it.
