How hard is calculus based physics without having taken high school or introductory physics?

It is not necessary, but very helpful.

Most calculus based physics classes are going to assume you have taken an algebra or trigonometry based course before.

You might struggle a little, but if you are willing to work hard then you will do fine.

That depends. Do you want to do electromagnetism? Or do you want to do classical mechanics?

If it's the former, then I'm afraid there's no way you can get by without a more advanced knowledge of calculus (assuming Calc 1 is stuff like Taylor polynomials, limits, simple integration and differentiation, etc). You'd need things like integration by parts, trigonometric substitution, and multiple integrals (surface integrals, volume integrals, cylindrical coordinates, polar coordinates, spherical coordinates) for calculating things like E/M fields inside a current coil, etc. Doing without high school physics would make matters even worse, since they'd probably assume you know things like basic electromagnetism.

If it's the latter, you have a better chance at doing well, given that the calculus based portions of the course wouldn't be too hard, probably only simple differentiation and integration. But they probably would require you to know things like E=1/2mv^2 and d=vit+1/2at^2, so the theoretical portion wouldn't be easy at all for you. Nevertheless, it's doable if you're willing to work hard enough.

Your courses might be easier than mines though, given that my courses are Engineering physics courses and are probably harder than what you'd get in other faculties.

You might want to at the very least read over a few basic physics things, like what Newton's laws are and the concepts of momentum, energy, etc. Since a lot of examples in Calculus 1 are based on mechanics, you should do well in that part of the course, which is also somewhat intuitive.

Where I'd be worried is electromagnetism, especially if you have no prior knowledge of Physics. Some of the concepts of fields, potentials, and circuits can be foreign to someone who's never seen physics, and as someone else said, the math is going to be far more advanced than just Calculus 1 in a calculus based physics course. You definitely would want to read at least about those concepts, and especially read up on things like Coloumb's Law, Faraday's Law and Ohm's law. Obviously the course will go more into it than that though, and will probably cover things like Maxwell's Equations.

At the very least, you would definitely want to be taking Calculus 2 at the same time as this course, since you will definitely need to know integrals to do the things in this class. However, I'm going to warn you that if you don't know the math AND the physics, you're going to really be in trouble. If you are trying to figure out what an integral is while not knowing what Coloumb's Law is, you're going to really struggle. Depending on what school you are going to, most likely this course could even include some math topics from Differential Equations and Multivariable Calculus, such as divergence, curl, and spring oscillation differential equations.

You could probably do it if you are picking up some of the Calc 2 as you go, but without a general idea of the Physics knowledge, I doubt you'll do well in that class if you are hung up on the math at the same time. You might want to finish Calc 2 first.