Can't stop fiddling!

As my profile information indicates, I am a total newbie to playing the fiddle. The fiddle has totally consumed me as I expected it would. For the first few months I was using a rental until I made sure it was something I was going to stick with and then I finally purchased my first fiddle back in March and haven't looked back. I practiced a lot with my rental and when I bought my own I started practicing even more. I feel like a junkie who can't put the pipe down when it comes to my fiddle. I usually put in 3-4 hours per day (sometimes more, especially on weekends). The main reason I stop playing is because it's too late and I don't want to wake my neighbor (I live in a duplex), or it's because my finger tips are so sore and my arms are so tired that it hurts to play the fiddle. I have to practice in shifts because of this and every time I have to put the fiddle down I become aggrevated that I have to stop simply because my fingers hurt or my arms are too tired to hold the fiddle properly. 

I really wish I had started playing the fiddle earlier in life. Another hobby of mine is playing hockey. I've been playing for 17 years now, but I didn't start until I was a young adult. One thing I've noticed is that players who started when they were very young, but have the same amount of playing experience or less than me, are often much better hockey players than I am and I think it has a lot to do with the fact that they started playing when they were very young. Granted they also play a lot more hockey than I do, but this still has me concerned about the fiddle because it makes me wonder if I will see the same thing as I do in hockey. I would love to be a good enough fiddler to play the difficult songs I love to listen to, but I fear my late start in life on the fiddle will limit how well I'm able to develop my playing skills, even after several years of experience. I know I won't get good overnight by any means, but I hope that with enough practice I'll be good enough to hold my own at a jam session and maybe even be good enough to be in an actual band playing the fiddle. Only time will tell if my practice will pay off enough to make this happen. For now, though, I'm content with being able to practice and get better so that one day I will be able to play with others who know what they're doing on their instruments. That being said, I can't wait until I know how to play my fiddle!