You want to use sudo but are worried about granting too many people too many
Good! You should be worrying about security.
While using sudo is much more secure than not using it, the default settings may be greatly improved.
Take the time to learn a bit about sudo itself and the /etc/sudoers file.
In particular, learn that in most cases you should not be using the ALL=(ALL) ALL specification!
Yes, that will work, but it’s not even remotely secure.
The only difference between that and just giving everyone the root password is that they don’t know the root password.
They can still do everything root can do. sudo logs the commands it runs, but that’s trivial to avoid by using sudo bash.
Second, give your needs some serious thought.
Just as you shouldn’t be using the ALL=(ALL) ALL specification, you probably shouldn’t be managing users one by one either.
The sudoers utility allows for very granular management and we strongly recommend using it.
man sudoers provides a wealth of material and examples, especially the section on preventing shell escapes.
sudoers allows for four kinds of aliases: user, runas, host, and command.
Judicious use of them as roles or groups will significantly reduce the maintenance burden.
For instance, you can set up a User_Alias for BUILD_USERS, then define the machines those users need to run on with Host_Alias and the commands they need to run with Cmnd_Alias.
If you set a policy to only edit /etc/sudoers on one machine and copy it around to all relevant machines periodically using scp with public-key authentication, you can set up a very secure yet usable system of least privilege.