How rewarding learning Elisp can be?

How rewarding learning anything is?

how exactly can i improve my emacs experience if i learned elisp?

You can program Emacs to do stuff you want it to do, the way you want it to do. It is like asking how can I improve my computer usage if I learn a programming language.

How can I become a more “proficient” Emacs user?

Just use it. It is that simple.

I also suggest being patient and looking up the documentation and the built-in help whenever you have a problem.

Don’t fall into some of the common pitfalls I see people here do often:

Don’t insist on super customized Emacs according to your liking from the early beginning. Be patient, and do stuff Emacs way, or live with some more irritated moments, and leave minutiae customization for later, when you are more familiar with Emacs. Emas is to be customized by each and every one and bent to your needs and personality, but it may take some deeper knowledge of Emacs in some areas, so to save yourself some frustration and lots of time, proceed gently at customizing Emacs in the beginning. This will probably be a somewhat controversial tip for some people, but what I mean is, prefer to do the work you have to be done, over how you do it in Emacs. In the beginning, bend yourself around Emacs, and keep in mind, that the more you are used to Emacs, it will be easier and faster to bend Emacs around you.

Yes, there are many tips and stuff accumulated on the Internet, but you can not learn everything at once. With the amount of information available, it is probably better to concentrate on using Emacs to do your work, and focus on exploring only those topics that give you the most in terms of how you perform your work; for example to solve some very irritating thing or to automate something, etc. You don’t have to start using Emacs for everything immediately, let it come with the time as you are getting more and more used to Emacs. What might be a killer feature for someone, might not be a killer feature for yourself.

Being popular does not always mean very good. Popularity goes in waves. You can spend all your time just re-learning “popular” frameworks and things. Instead of focusing on which package you should use for this or that; choose one and just use it. Once you find some true shortcoming that irritates you, than look for another package or for tips on how to change it, etc. If you are using Doom, I am quite sure they have already included something that is good for most people. I personally just use whatever is default in Emacs and have had no issues with it; for like 25 years now, or something there and counting.

Looks defunct

It is. You can still use it. There is an implementation that byte compiles with just some minor warnings but EIEIO (a simplified CLOS version) is what is available in Emacs as built-in and there is much more available resources to learn from if you want to use an object-oriented system in Emacs.

if you have to understand (exactly) what is happening under the hood of an abstraction in order to use it correctly, then it’s not a good abstraction.

Nobody says you have to understand it exactly, but you have to have an understanding of what is going on. You also have to have an understanding of the abstraction itself, of what are you using. Have you even read the use-package documentation? I don’t recall ever that use-package was meant to hide away the Emacs itself, but to help you write more structured setup. If you don’t like it don’t use it, it is not harder than that. I don’t use it myself. But your problem is neither Emacs nor use-package. I personally can’t care less if you use or not, but you are now blaming the missunderstanding on external factor. Lots of people are using use-package and find it useful. Perhaps you should reflect over if it is the software or the user :-). I don’t mean to be impolite or arrogant, but sometimes things feel difficult when we are not ready for them. Nobody promised that use-package should be trivial, albeit I personally don’t think it is very difficult tbh.

You have two packages: X and Y. You want to do something with Y if X is loaded, but sometimes you want to just “jump” into Y without loading X, and than wonder why your X based setup will not load. In Emacs your vanilla option is to always load X whey you load Y, or cook your own thing. In your particular example, your option is to load project.el whenever you load Magit, it is not harder than so. Just require package.el in your Magit configuration.

requires me to more-or-less just macro-expand all of my declarations to see what they are actually doing.

Welcome to Lisp :-). You can compare that to C++ where you more or less have to look at assembly output if you care about efficiency and what your compiler does behind your back. I think you should actually be thankful you can do that. Try to do that with Python, JS or you name it.

The fact that so many people struggle with it gives credence to this as well.

I would rather care how many people do not struggle when looking at the big picture of how many people actually use use-package.

Frankly, Emacs is complex and complicated.

Sure, but so is any piece of software that does non-trivial tasks. If you find Emacs too complex and complicated, to the limit that it causes too much frustration and time loss, don’t use it, nobody is holding your hands.

It has lots of legacy baggage and idiosyncrasies

Sure, I agree. I personally use to say that Emacs is a hack over a hack. And I am really surprised how well it runs and does what it does, how hacked together the source is. But is to expect of 50 year old software. It is developed by many people with different backgrounds and different goals, mostly by hackers who just wanted to bend Emacs to do their own thing. It is a hackable editor made for hackers.

I would expect people to get their with-eval-after-loads wrong or their keybinding syntaxes wrong

I wouldn’t. Mostly because those two particular things are incredibly simple to get correct.

use-package seems to hurt as much as it helps

I think it is individual. What you did wrong in your example is that you haven’t required project.el in your Magit configuration. Do it and your problems will be gone. Simple. Nothing wrong with use-package.

I can also tell that you can just use with-eval-after-load and mode hooks to achieve the same lazy setup if you prefer not to use use-package. I don’t think it is that hard; I have done it myself and I was not even an experienced Emacs user or Emacs Lisp programmer at that time.

I can understand your frustration; I have been there myself, and my best tip is: start reading the documentation and built-in help. It is a best first step towards understanding Emacs or user-package one can take.

I think your problems are due to not understanding how Emacs works under the hood. To me, the only problem with use-package is that it hides those details and makes people learn a DSL that use-package is, instead of learning simpler mechanisms of with-eval-after-load and hooks which are used to implement the functionality of use-package under the hood. In past years I have seen many questions here on Reddit and on SX related to use-package because people don’t understand Emacs under the hood. But that is about it; I see no other drawbacks than that.

Personally, I have stopped using use-package after about 2 ~ 3 years ago. Been using it for a few years before that. The reason why I abandoned it was actually because I wanted to learn how such a thing was implemented and how Emacs worked under the hood. Another reason was that use-package was not included in Emacs back in time. However, I am not going back to use-package, just because it got included. Partially I am too lazy to rework my setup again, and partially I personally don’t need it.

To be clear, I don’t agree that use-package is wrong about anything or poorly written or anything like that. On the contrary, use-package has its very good points. One of them is that everything related to a single package is condensed into one place. At least configurations that are built around the use-package tend to be well structured. That part itself is a very good reason to use use-package, unless you have the very good discipline or come up with a system to organize your init file.

There is no built-in automatic way; you will have to backup your old values any way you want; create a bunch of new symbols, put them into a hash table, a list, or whichever else strategy you prefer.

You don’t need to move files around just to bulk rename them. If you want to do it, sure, you can, but you don’t have to.

In your top level directory:

1. C-u C-x d (alternatively C-u M-x dired)

You will be in minibuffer now. Be sure dired listing switches have -l and -R option; -l is probably already there among anything else you might use; just type space and add -R at the end and type Return (Enter)

Now you will have a Dired buffer with al the subdirectories and files in them listed in the same buffer.

2. C-x C-q in Dired to switch to WDired mode (I have bound it in my Emacs to C-S-r for easier typing)

You are now in “writeable dired” mode where you can edit all file names as if it were an ordinary text buffer. You can also use replace-string for example to replace a part of the name in all filenames at once, regex-replace, etc. You can do it manually, or whatever else you would do in a text buffer.

3. C-c C-c to save your changes when you are done; all files will be updated.

I think it would be a better task for just the CLI

No, it wouldn’t :). You can certainly do it, but it is frankly, easier with Emacs only.