This box is a 11 year old dell, and I’d not like to have only 4gb of ram.
On another PC I have that is 12 years old, I recently stole 4gb of it’s 8gb of ram, and for many tasks the difference is not noticeable, however on others it sure is! (I use that machine far less since I stole 4gb of it’s ram; its not as nice to use anymore!)
If you have less ram, you need to be far more careful with your tasks, the order you do tasks, how many co-exist at the same time etc… so as to not waste the limited ram unless you’re willing to suffer degraded performance.
You’re also mentioning server functions; so to me the Lubuntu or a GUI means you are already wasting resources just to have a GUI.
If however you’re only worried about boot up time, no it won’t make that much difference (I believe it would make more difference to performance during operation)